Remote camera augmented reality system

ABSTRACT

There is disclosed a system for remote tracking of augmented reality media through a real-world environment. The system may use an augmented reality platform incorporating a camera and motion sensors to move about a physical space while an augmented reality object or characters are superimposed within that physical space. The augmented reality platform may move intelligently to keep the augmented reality media within frame as it moves or is adjusted. The augmented reality platform may present a basis upon which to build upon for superimposing augmented reality elements onto augmented reality media and the augmented reality platform as they move about within a space together. The platform may be used, for example, in a racing or platforming style game or experience.

RELATED APPLICATION INFORMATION

This application is a continuation of U.S. patent application Ser. No.16/294,536 filed Mar. 6, 2019 entitled “Remote Camera Augmented RealitySystem,” now U.S. Pat. No. ______ which, like this application, claimspriority from the following provisional patent applications, theentirety of which are incorporated by reference:

U.S. provisional patent application No. 62/639,397 filed Mar. 6, 2018and entitled “Third Person Perspective Augmented Reality System”; U.S.provisional patent application No. 62/639,398 filed Mar. 6, 2018 andentitled “Remote Augmented Reality Camera for Image Processing andMapping”; U.S. provisional patent application No. 62/639,400 filed Mar.6, 2018 and entitled “Augmented Reality Camera Control for CombinedVirtual and Augmented Reality Environments”; U.S. provisional patentapplication No. 62/639,402 filed Mar. 6, 2018 and entitled “ImageDetection-Based Progression System”; and U.S. provisional patentapplication No. 62/639,404 filed Mar. 6, 2018 and entitled “InertialMeasurement System for Virtual Avatar and Real World Interaction”.

NOTICE OF COPYRIGHTS AND TRADE DRESS

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. This patent document may showand/or describe matter which is or may become trade dress of the owner.The copyright and trade dress owner has no objection to the facsimilereproduction by anyone of the patent disclosure as it appears in thePatent and Trademark Office patent files or records, but otherwisereserves all copyright and trade dress rights whatsoever.

BACKGROUND Field

This disclosure relates to augmented reality and more particularly toaugmented reality through images provided by a physically distinctdevice that enable various interactions within augmented reality andvirtual reality environments.

Description of the Related Art

There exist various implementations of augmented and virtual reality.For example, various head mounted displays have enabled virtual realityand augmented reality experiences when worn on a user's head and inconjunction with appropriate software. In more recent incarnations,augmented reality experiences or toys have provided systems reliant uponmobile devices (e.g. smart phones) that use their included cameras tofilm reality and to superimpose objects or characters within thatreality. More complex systems like Microsoft® HoloLens® enable augmentedreality and mixed reality experiences for wearers that provide some or agreat deal of interaction with the augmented reality components of thoseexperiences such as voice communication or live video-like feeds forholographic conferences.

In a relatively unrelated field, camera drones have become commerciallyviable at a consumer level. There exist numerous drones, typicallyflying quadcopter drones, that include or have mounts for cameras thatmay film as they go, or film and stream directly to a controller ormobile device. These types of drones are used, for example, infilmmaking, by vacationing individuals, in events like drone racing, andindustrial applications for moving or filming in places whereindividuals cannot reach or see.

DESCRIPTION OF THE DRAWINGS

FIG. 1 is an overview of an augmented reality racing system.

FIG. 2 is a block diagram of an exemplary computing device.

FIG. 3 is a functional block diagram of an augmented reality racingsystem.

FIG. 4 is a flowchart of remote camera augmented reality and raceprogression tracking.

FIG. 5 is an example display screen of a mobile device showing remotecamera capture and race progression tracking.

FIG. 6 is an example display screen of a mobile device showing remotecamera augmented reality and race progression tracking.

FIG. 7 is an overview of a sample race progression tracked using raceprogression tracking.

FIG. 8 is a flowchart of adjustment of color and lighting for augmentedreality objects to correspond to real world objects.

FIG. 9 is a flowchart of augmented reality and virtual reality remotecamera tracking.

FIG. 10 is an example of augmented reality remote camera tracking.

FIG. 11 is an example of augmented reality into virtual reality remotecamera tracking.

Throughout this description, elements appearing in figures are assignedthree-digit reference designators, where the most significant digit isthe figure number and the two least significant digits are specific tothe element. An element that is not described in conjunction with afigure may be presumed to have the same characteristics and function asa previously-described element having a reference designator with thesame least significant digits.

DETAILED DESCRIPTION

The combination of augmented reality technology with remote cameradrones enables several new applications including: gameplay that isreminiscent of traditional video racing games but that involves or takesplace substantially within reality: playing of platformer-style gamesthat strongly resemble traditional platformers but that take placewithin reality: or experiences that enhance or broaden the scope of orotherwise play on the space around a player or user to expand it throughaugmented reality that is experienced from the perspective of the droneor other AR platform.

The use of augmented reality with a remote camera is novel in that itintroduces numerous control and operational issues that must beaddressed. For example, one must decide which data is relevant to theaugmented reality operation by a remote control (e.g. a mobile device)to enable the creation and accurate representation of augmented realityobjects from a remote host (which may also be the same mobile device).With the opportunity to integrate multiple sources of information (e.g.visual data and motion data), the environments may be more effectivelymapped initially and monitored during augmented reality operation. Theactual camera, provided by the augmented reality drone or otherplatform, must be controlled, either manually or automatically, tomonitor the augmented reality media being superimposed over the reality.To increase immersion, the barrier between augmented reality and realitymust be blurred as much as possible through practical as well assoftware-based advancements.

These and other problems are addressed as described below.

Description of Apparatus

Referring now to FIG. 1, an overview of an augmented reality racingsystem 100 is shown. The system includes a series of mobile devices 110and 112, AR platforms 120 and 122, and a server 130, all interconnectedvia a network 110.

The mobile devices 110 and 112 are computing devices (e.g., as shown inFIG. 2) operated by users 115 and 117. The users 115 and 117 are not apart of the system, but are shown merely to be representative of usersoperating the mobile devices 110 and 112. Many potential users canparticipate using associated mobile devices at any time.

The mobile devices 110 and 112 are used to view a display having imagedata captured and provided to the mobile devices 110 and 112 by the ARplatforms 120 and 122. The mobile devices 110 and 112 may also have oneor more IMUs (inertial measurement units), communication capabilities,and processing capabilities capable of generated augmented reality orvirtual reality environments. The mobile devices 110 and 112 may alsoincorporate physical controls (e.g. buttons and analog sticks) forreceiving control input from the users 115 and 117. Alternatively,on-screen controls (e.g. capacitive or resistive on-display virtualbuttons or joysticks) may operate as controls for the AR platforms 120and 122. The mobile devices 110 and 112 may also have speakers and/orthe capability to integrate with headphones to provide sound fromassociated augmented reality or virtual reality software operating onthe mobile devices 110 and 112.

The AR (augmented reality) platforms 120 and 122 are mobile platformsincluding at least one camera for capturing video image data andtransmitting that image data to the mobile devices 110 and 112. The ARplatforms 120 and 122 may be “dumb” in the sense that they include noneor very limited computing capabilities. Alternatively, the AR platforms120 and 122 may incorporate a computing device (e.g., as shown in FIG.2) for performing various operations. The AR platforms 120 and 122 arealso capable of being controlled using the mobile devices 110 and 112based upon control input received from the mobile devices 110 and 112input by users 115 and 117. The AR platforms, therefore, include atleast one camera and potentially include one or more IMUs as well asmotors and a locomotive system.

As used herein, locomotive system means a system for moving the ARplatform 120 or 122 from one location to another. The locomotive systemincludes a motor or other motion-generation element and a component thatmoves in response to the motion-generation element to cause the ARplatform 120 or 122 to move from one location to another. Examples oflocomotive systems include a motor and one or more wheels which move anAR platform similar to a remote-controlled car, a motor and at least onerotor of an aerial drone which flies about in the air and may becontrolled by a remote control or mobile device, a motor and propellerof an AR platform remote-controlled ship or submarine which moves aboutin water. Other locomotive systems are also envisioned.

The AR platforms 120 and 122 use their respective locomotive systems tomove from location to location capturing updated image data and relayingthat image data back to mobile devices 110 and 112. The mobile devices110 and 112 may operate upon that image data to generate augmentedreality objects or virtual reality environments and to make a driver(e.g., users 115 and 117) aware of the overall environment of the ARplatforms 120 and 122 so that the driver can continue providing controlinput that may be used to instruct the locomotive systems in continuingto move the AR platforms 120 and 122 from place to place.

The server 130 may or may not be present in all situations.Specifically, the present concepts are envisioned to be used inconjunction with both augmented reality and virtual reality games andother immersive experiences that may be single-player or multiplayer.Multiplayer games and experiences may be “hosted” by one of theparticipants. However, matchmaking, match tracking, authentication andother capabilities may be handled by the server 130 when multiple users115 and 117 experiences the associated augmented reality or virtualreality together. In cases in which experiences or games aresingle-player, the server 130 may or may not be used at all.

The server 130 is shown as a single computer, but may in fact be manycomputers spread across numerous locations. The server 130 may be one ormore virtual servers, operating in the midst of a larger, physicalserver.

The network 110 is a communications medium for interconnecting thevarious components of the system 100. The network 110 may be or includethe internet, wireless or wired networks, local or wide-area networks,and/or telephone networks. The network's primary purpose is to enablecommunication between the various components of the system 100 to beshared so that augmented reality or virtual reality experiences or gamesmay take place (e.g., enable image data captured by an AR platform 120to be transmitted to mobile device 110 and, likewise, to allow controldata generated by the mobile device 110 to be transmitted to AR platform120).

Turning now to FIG. 2, a block diagram of an exemplary computing device200 is shown. As shown in FIG. 2, the computing device 200 includes aprocessor 210, memory 220, a communications interface 230, along withstorage 240, and an input/output interface 250. Some of these elementsmay or may not be present, depending on the implementation. Further,although these elements are shown independently of one another, eachmay, in some cases, be integrated into another.

The processor 210 may be or include one or more microprocessors,microcontrollers, digital signal processors, application specificintegrated circuits (ASICs), or a systems-on-a-chip (SOCs). The memory220 may include a combination of volatile and/or non-volatile memoryincluding read-only memory (ROM), static, dynamic, and/ormagnetoresistive random access memory (SRAM, DRM, MRAM, respectively),and nonvolatile writable memory such as flash memory.

The memory 220 may store software programs and routines for execution bythe processor. These stored software programs may include an operatingsystem software. The operating system may include functions to supportthe communications interface 230, such as protocol stacks,coding/decoding, compression/decompression, and encryption/decryption.The stored software programs may include an application or “app” tocause the computing device to perform portions of the processes andfunctions described herein. The word “memory”, as used herein,explicitly excludes propagating waveforms and transitory signals.

The communications interface 230 may include one or more wiredinterfaces (e.g. a universal serial bus (USB), high definitionmultimedia interface (HDMI)), one or more connectors for storage devicessuch as hard disk drives, flash drives, or proprietary storagesolutions. The communications interface 230 may also include a cellulartelephone network interface, a wireless local area network (LAN)interface, and/or a wireless personal area network (PAN) interface. Acellular telephone network interface may use one or more cellular dataprotocols. A wireless LAN interface may use the WiFi® wirelesscommunication protocol or another wireless local area network protocol.A wireless PAN interface may use a limited-range wireless communicationprotocol such as Bluetooth®, Wi-Fi®, ZigBee®, or some other public orproprietary wireless personal area network protocol. The cellulartelephone network interface and/or the wireless LAN interface may beused to communicate with devices external to the computing device 200.

The communications interface 230 may include radio-frequency circuits,analog circuits, digital circuits, one or more antennas, and otherhardware, firmware, and software necessary for communicating withexternal devices. The communications interface 230 may include one ormore specialized processors to perform functions such ascoding/decoding, compression/decompression, and encryption/decryption asnecessary for communicating with external devices using selectedcommunications protocols. The communications interface 230 may rely onthe processor 210 to perform some or all of these function in whole orin part.

Storage 240 may be or include non-volatile memory such as hard diskdrives, flash memory devices designed for long-term storage, writablemedia, and proprietary storage media, such as media designed forlong-term storage of data. The word “storage”, as used herein,explicitly excludes propagating waveforms and transitory signals.

The input/output interface 250, may include a display and one or moreinput devices such as a touch screen, keypad, keyboard, stylus or otherinput devices.

FIG. 3 is a functional block diagram of an augmented reality racingsystem. The functional block diagram includes a mobile device 310, an ARplatform 320, and a server 330, which may be the mobile device 110, theAR platform 120 and the server 130 of FIG. 1. There may be many mobiledevices, like mobile device 310, and many AR platforms, like AR platform320. However, since their functions will largely be the same, only theindividual mobile device 310 and AR platform 320 are shown. In addition,underlying components that are not particularly relevant to thisdiscussion are omitted such as operating system software, communicationsstacks and protocols, and network routers and connections.

The mobile device 310 may be a traditional mobile phone or smart phone.The mobile device 310 may be a tablet computer or laptop computer. Themobile device 310 may be a handheld gaming platform. Though described asa “mobile device” the device 310 may be fixed as a traditional computeror traditional gaming console. Virtually any computing device ofsufficient processing power and suitable inputs may operate as themobile device 310. Though it may be preferable for the mobile device 310to be movable, it is not required.

The mobile device 310 includes AR/VR software 311, object recognitionsoftware 313, a display 314, an IMU (inertial measurement unit) 316, oneor more camera(s) 318 and a user interface/control system 319. Some ofthese elements, for example, the object recognition software 313 and IMU316, may be optional and may or may not be included in the mobile device310.

The AR/VR software 311 is software for generating augmented reality orvirtual reality environments. The distinction between AR (augmentedreality) and VR (virtual reality) that is relevant for this disclosureis that AR incorporates some element of reality whereas VR is anentirely computer-generated experience. AR may, for example, be a videoview of the real world through a camera, recreated on a display, withalterations to that video. The alterations may be as simple as filters(e.g. the addition of large eyes, or an unusual hat, or sparkleshovering near a person's head) when the video is seen. The additions arecomputer generated using AR/VR software 311, and the remainder of theimage is captured by the camera(s) 318 and reproduced substantially asit appears in reality.

As used herein, the phrase “augmented reality media” meanscomputer-generated content that is or may be overlaid onto images ofreality (e.g. digital video) to superimpose that content into thoseimages as if it were present in reality. Augmented reality media couldinclude signs, characters, vehicles, toys, plants, partial environments,and virtually any other computer-generated images that may besuperimposed over images of reality.

Other augmented reality elements may be characters with whom anindividual can interact (e.g. speak with or shake hands with or sitnear), real individuals that are being projected into a place in animage of reality (this is often called “mixed reality”), a location, asign, vehicles, components of vehicles or other objects, or otheraspects that are not present in reality but that are generated by theAR/VR software 311.

Virtual reality is an environment into which one may “look” or withwhich a user may interact that is entirely computer-generated. Thesetypes of environments often rely upon so-called “game engines” becausegame engines are particularly good and quick (when coupled with theright hardware) at generating believable three-dimensional locations.Virtual reality is typically associated with some form of headwear (e.g.a head-mounted display) which wholly covers the entire field of visionof a wearer. Then, a wearer's head is “tracked” as he or she turns ormoves about in a space, and the virtual reality environment is updatedto reflect that. However, virtual reality need not require a head worndisplay. In some cases, the display device (e.g., a mobile device) mayact as a “portal” into the virtual world through which a user may look.Instead of tracking the user's head, movement of the display device maybe tracked to update the direction within the virtual environment thatan associated virtual camera is looking. The images of the virtualreality environment on the display device may be updated accordingly.

The AR/VR software 311 is software that generates augmented realityand/or virtual reality, depending on the particular implementation. ThatAR/VR software 311 may rely upon several inputs. Those inputs mayinclude IMU data (e.g., positional, locational, and orientational data),camera images (e.g., RGB images or associated video captured by adigital camera), infrared or LIDAR images (e.g., spatial images havingdepth or approximating depth) so that the software 311 can have anunderstanding of the spatial area in which it is operating, and controlinput data which may be external control of the location of theassociated cameras and IMUs (e.g., on the AR platform 320, as discussedbelow). The AR/VR software 311 may rely upon information captured by theelements of the AR platform 320 to generate augmented reality objects ora virtual reality environment.

The object recognition software 313 may be a part of the AR/VR software311 or may be separately implemented. Object recognition software 313may rely upon camera data or specialized cameras (such as infrared andLIDAR) to perform object recognition to identify objects within imagesof an environment. In the most basic version of the object recognitionsoftware 313, the software operates to identify barcodes, QR codes orother machine-readable elements. Those codes may be used to markrelevant portions of reality or other objects (e.g., other AR platforms320) in reality. This will be discussed more fully below.

In a more complex variation, the object recognition software 313 mayoperate using trained neural networks to identify specific objects orobject types in substantially real-time. Object recognition software 313may be able to match the same object each time it passes by or each timeit is detected in image data from a camera, or to identify the samethree-dimensional shapes or set of shapes each time they pass usingspecialized cameras. The object recognition software 313 may rely uponboth traditional images and specialized images including depthinformation to perform object recognition.

The display 314 is a computer display of sufficient size and quality fordisplay of image data captured by the camera(s) 318 (or, more likely,the camera(s) 323). The display 314 may be capacitive or resistive, andmay incorporate software controls or readouts of relevant informationabout the mobile device 310 or AR platform 320. The display is capableof real-time or nearly real-time display of image data transmitted tothe mobile device 310 from the AR platform 320 and of overlay ofaugmented reality (or full replacement with virtual reality) of imagedata as directed by the AR/VR software 311 without significant slowingof frame rates of the display (approximately less than 60 ms delay).

The IMU 316 is an inertial measurement unit. The IMU 316 may not benecessary in the mobile device 310, but may be used, for example, as apart of a control scheme. For example, in a driving or flying augmentedreality game, movement of the mobile device 310 may translate intosteering. Movement or orientation of the mobile device 310 in certaindirections within a virtual reality environment may instruct anon-screen avatar to move forward, backward, jump, or the like.

The camera(s) 318 likewise may not be required in the mobile device 310.However, if present, camera(s) 318 may provide a secondary source ofinformation for the object recognition software 313. The mobile device310 is typically held at a waist height or in front of a user's eyes.So, it may offer a different (e.g., higher or lower) vantage point toenable object recognition to operate more efficiently. The camera(s) 318may also enable tracking of the AR platform 320, either approximately oraccurately, from a vantage point outside of the AR platform 320 itself.The camera(s) 318 may be traditional digital video cameras, but may alsoinclude infrared, LIDAR or other depth-sensing cameras.

The user interface/control system 319 is used to control the locomotivesystem 324 of the AR platform 320. The user interface/control system 319may be or include the IMU 316, as discussed above. However, the userinterface/control system 319 may be a more traditional controller orcontroller-style system including a series of buttons and control sticksor a keyboard and/or mouse. The user interface/control system 319 mayalso include software-based controllers such as on-screen buttons andmovement systems that are shown on the display 314.

The AR platform 320 is a mobile and remote augmented reality deviceincorporating at least one camera. It is an “AR” platform in the sensethat it captures video and/or IMU data that is used to generate anaugmented reality by the AR/VR software 311 in the mobile device 310.Unlike typical augmented reality systems, the camera(s) 323 used forthat augmented reality are remote, operating upon the physicallyseparate AR platform 320, rather than on the mobile device 310. Examplesof AR platforms 320 include remote-controlled cars, remote-controlledairplanes, helicopters and drones, and remote-controlled ships andsubmarines. Other examples are possible.

The AR platform 320 includes a control interface 321, one or morecamera(s) 323, a locomotive system 324 and an IMU 326. Other elementsmay be present in some cases.

The control interface 321 is for receiving instructions from the mobiledevice 310 (e.g. the user interface/control system 319) and passingthose along to the locomotive system 324 to cause the AR platform 320 tomove as directed by the mobile device 310.

The camera(s) 323 are cameras attached to the AR platform 320 to captureimage data to be transmitted to the mobile device 310 and so that a userof the mobile device 310 may see and control the movements of the ARplatform 320. The camera(s) 323 may be traditional digital videocameras, but may also include infrared, LIDAR or other depth-sensingcameras.

The locomotive system 324 is discussed above. The locomotive system 324is how the AR platform 320 moves from location to location. Thelocomotive system may be a set of wheels and a motor for aremote-controlled car-like AR platform 320. The locomotive system 324may operate on a rail system in some cases or may be a helicopter ordrone.

The IMU 326 is one or more inertial measurement units to generatemotion, positional, and orientational data for the AR platform 320. Thedata generated by the IMU 326 may be passed to the AR/VR software 311 toupdate augmented reality object(s) or virtual reality environment inwhich the AR platform 320 is operating.

The server 330 includes a game server 332 and a database server 334.Numerous other elements may be present in different situations, butthese two general components are present in most implementations thatrely upon or use a server like server 330. As indicated above, theserver 330 itself may be optional. To the extent a server 330 is used atall, it may have these components.

The game server 332 is software for matching players with one another,for performing authentication to enable players or users to access theserver, for operating or maintaining an augmented reality or virtualreality in synchronicity between multiple participants in the augmentedor virtual reality so that the same augmented reality objects are seenby all participants, and for tracking progress or position within theaugmented reality or virtual reality environment. If there are winconditions or other conditions, the game server 332 may track those aswell. It is called a “game server” primarily because the AR/VR software311 typically operates from a basis of a game engine. So, the gameserver 332 is likewise traditionally used for and with games. The gameserver 332 may also generate statistics or other information about theaugmented reality or virtual reality it is serving to participants. Thegame server may store a copy of the augmented reality or virtual realitycomponents and may enable the AR/VR software 311 to be updated ormaintained.

The database server 334 is to store authentication credentials,statistics or other information about participants in the augmentedreality or virtual reality, and any other database information requiredto operate augmented reality and virtual reality environments includingmultiple participants.

Description of Processes

Referring now to FIG. 4, a flowchart of remote camera augmented realityand race progression tracking is shown. The flow chart has both a start405 and an end 495, but the process may take place many times or manytimes for different AR platforms 320 (FIG. 3) at a time or at differenttimes. In general, the processes on the left take place on an ARplatform, while the process on the right take place on an associatedmobile device. Though, in some cases, some of the processes may takeplace, at least in part, on an AR platform or the mobile device or both.

Following the start 405, the process begins with the capture of imagedata 410. As used herein “image data” means traditional digital videoimage data. However, image data may also include depth data (e.g. datacaptured by infrared, sonically, or through other methods) to create an“image” of the surroundings. The image data is captured by the camera(s)323 on the AR platform 320. As indicated above, image data may also becaptured in some cases by the mobile device 310 camera(s) 318, but thisis not required.

This image data is captured at 410 on a substantially continuous basis.For example, as the AR platform 320 (FIG. 3) moves from place to place,the image data changes because the camera(s) 323 capture new image datafrom that new location. Preferably, that image data is streamed orotherwise transmitted to the mobile device 310 (FIG. 3) as quickly aspossible. Dropped frames of video or partially dropped frames may simplybe ignored, with later frames taking their place, because it is muchmore important to continue providing a substantially live view from thecamera(s) 323 than to make sure every frame is properly included. Imagedata is captured using at least one AR platform 320 camera(s) 323, butmay also rely upon or be integrated with image data from other ARplatforms 320 operating in the same real world environment or virtualenvironment.

Next, motion data may be captured at 420. This is shown as an optionalstep because it may not be present in all implementations, but it ispossible to incorporate the motion data for other purposes discussedbelow. Motion data is generally data generated by the IMU 316 thatincludes data that may be used to estimate speed, estimate motion liketurning or movement vertically relative to gravity, and detect eventssuch as flipping over, being hit, running into things, and the like. Allof the foregoing may be captured by the IMU 316 for the AR platform 320.As with image data, the motion data is captured using at least one IMU316 on one AR platform 320, but multiple AR platforms 320 may capturemotion data.

Next, the image data and motion data (if captured) may be transmitted at430 to the mobile device. As indicated above, this happens as quickly aspossible and may effectively be “streamed” from the AR platform 320 tothe mobile device 310. That image data and motion data may come frommultiple AR platforms 320 or only from one.

The AR/VR software 311 may update the display/status at 440. When imagedata is first streamed to the mobile device 310, this update thedisplay/status at 440 may merely be showing the associated capturedimage data on the display 314 of the mobile device 310. Thereafter, theupdate the display/status 440 may involve updating the image to show themost recently-captured image data.

In addition, the AR/VR software 311 may incorporate one or moreaugmented reality or virtual reality elements in response to the imagedata (and motion data) onto the display 314 when it is updated at 440.These updates may be to show augmented reality objects at a new orupdated location, to re-center or re-render a virtual realityenvironment, or to show responses to the status or image data. Forexample, if an augmented reality racing game is taking place, augmentedreality road signs may be updated to show a leader board with positionsof the various participants in the race. If a lap-marker was justpassed, it may be noted on the display by an augmented reality banner orin an augmented reality character reaction. If a character is in themidst of being controlled by a mobile device 310, then the character maybe updated to reflect the controls being provided to that character.

So, for example, a virtual character or virtual racing machine (e.g.cart, plane, etc.) may be superimposed over an actual image. Thesuperimposition may be whole (e.g. fully blocking any view of the ARplatform 320) or partial (only blocking some of the AR platform whileshowing other portions). The augmented reality characters or objectsoverlaid on top of the actual image data may react to the image dataand/or motion data. For example, if the AR platform 320 is movingquickly in a racing game, an associated character may respond by actingor being animated such that he or she appears to be windblown orotherwise concerned about the speed of the AR platform 320. Afterburnersmay fire up on an augmented reality plane. An augmented reality crowdmay react to movement of or other information about the real-world ARplatform 320. Other updates to the display/status at 440 may take place.

Several optional operations are shown here in elements 442 through 448.These operations may or may not take place in various implementations.

The first of these optional operations is to perform image processing at442 of the captured image data. This image processing may be performedby object recognition 313 and/or AR/VR software 311. The imageprocessing may be simple, such as selecting a frame or a series offrames and analyzing them to identify machine-readable elements. Thisprocess may be complex, such as spatial recognition, reliant uponimage-only data or image and depth data, to generate a map of thephysical area surrounding the AR platform 320 as it moves.

Depth data, in particular, whether it is estimated using images (e.g.stereography) or actual using depth sensors like infrared or LIDAR maybe helpful in the image processing. The image processing at 442 mayenable the mobile device 310 to construct a relatively detailedunderstanding of the three-dimensional environment in which the ARplatform 320 is operating at any given moment. That may be used fornumerous purposes, some of which are discussed herein.

The first of those purposes is to detect visual landmark(s) at 444. Asused herein, a visual landmark is an aspect of the physical world thatis detectable by the mobile device 310 using at least image datacaptured by the camera(s) 318 on the AR platform 320 and usable by themobile device 310 to identify the AR platform 320's relative locationwithin a three-dimensional space. Examples of visual landmarks includeQR codes, barcodes, particular colored objects, particular shapedobjects, and the detection and identification of repeats of the sameobject (e.g. the same chair has been passed three times). The subset ofvisual landmarks that are generated specifically for computer detection,such as QR codes, barcodes, particular colors or shapes of objects,particular light configurations or colors, and similar elements are“machine-readable elements”.

The visual landmark(s) enable the system to function at least somewhatindependently to operate as a game, and enable the system to generateand update its own area map 446. As an AR platform 320 or multiple ARplatforms move about in a physical space, they are continuouslycapturing image data about that space. Over the course of several turnsabout that space, potentially laps or flights throughout around thatspace, the AR platforms generate sufficient image data to inform objectrecognition 313 of the three-dimensional and visual contents of thatspace.

As a result, the mobile device 310 may generate/update an area map 446showing the general layout and makeup of the area. That map may besimple, and top-down showing objects overlaid with their images that arecaptured from within the image data or may be three-dimensional and mayeven be so complex as to operate as a virtual reality map or to be ableto be replaced with a virtual reality environment into which others maythen interact.

For example, the virtual reality map may be a map of a living room inwhich a series of flying drones are operating in a race. After threelaps, with three drones, the resulting three-dimensional map createdduring that process and associated image processing may generate a mapthat is sufficiently detailed that, later, a virtual drone may be flownwithin that same space, now recreated in virtual reality. The samecontrols may operate in both spaces (e.g. controls to control thephysical drones may be co-opted later to control the virtual drone inthe virtual reality environment of that virtual reality map).

In a more basic example, after several laps around a space usingremote-controlled AR platform 320, the layout and general physicalcomponents surrounding a race area may be sufficiently known such that atop-down view of a “map” of the race location may be generated anddisplayed on each participant in a racer's display 314 on theirassociated mobile device 310.

At the same time, this updating the area map 446 and updateddisplay/status may also track progression within an environment. Thedetected visual landmarks may be used, for example, for trackingpositions within a race and race completion or other victory conditions.As an AR platform 320 or group of AR platforms move about a space,relative location and/or progression may be tracked and updated at 440to determine progression within the competition. The visual landmarksmay provide guideposts as they are approached or crossed that inform theoverall system that various conditions for the AR platform 320 are beingmet.

The final optional update to the display/status 440 is to update thedisplay regarding motion data at 448. In cases in which motion data iscaptured at 420, that data may further inform the characters, augmentedreality objects or any virtual reality environment. For example, motiondata indicating a very tight turn for the AR platform 320 may cause anaugmented reality character sitting in or on the AR platform 320(digitally, not actually) to appear to hang on for dear life or tonearly fall out of the AR platform 320. The AR platform 320 crashinginto something (as detected by the IMU 316 may cause the AR object suchas an AR overlay car to appear wrecked or crashed or smashed, or acharacter riding in or on the car may fly out and climb back in whileappearing dizzy.

An IMU 316 may detect inversion of an AR platform 320 or that a flyingAR platform 320 has likely crashed. An on-screen augmented realityavatar may encourage a user to flip the AR platform 320 back over or mayanimate a fiery crash. This data may include flips, rollovers,inversions, bumps, crashes, reversing, bouncing, and various other typesof motion data. Augmented reality or virtual reality may be updated toreflect any of that motion data.

This may be further integrated with IMU data from multiple AR platforms320 being integrated. For example, if two AR platforms 320 both detectside-hits, one on its left side, the other on its right side atsubstantially the same time, then the display 314 of the mobile devices310 associated with each AR platform 320 may be updated to indicate thatthe two AR platforms 320 bumped into one another. Augmented realitycharacters or objects may be updated accordingly to reflect dings orhave characters animate annoyance. In this way, motion data frommultiple sources may be combined to inform augmented reality or virtualreality.

The process then determines whether tracking is complete at 445. If thetracking is complete such as when the race is over, the game is over, orthe character dies or fails (“yes” at 445), then the process may end at495.

If tracking is not complete (“no” at 445), then control input from themobile device 310 may be received at 450 by the AR platform 320. Thesecontrols may direct an on-screen character, movement of the AR platform320 itself, or some combination of both. The controls may operate tocause the AR platform 320 to move while not moving the character or theaugmented or virtual reality (e.g. adjusting the real-world camera foran augmented reality player character or other object).

Those control inputs may be transmitted at 460 to the AR platform 320.The control inputs may direct the movement of the AR platform 320 sothat it turns, moves vertically, beeps a horn, or otherwise changes itsmovement or location.

The AR platform 320 is adjusted at 470 in accordance with the controlinputs provided at 460. As a result, the image data changes and iscaptured anew at 410 and the process continues until tracking iscomplete at 445.

FIG. 5 is an example display screen of a mobile device 510 showingremote camera capture and race progression tracking. A real-world livingroom 570 is shown, near a kitchen from the perspective of an AR platform520. The living room includes a chair 550 and a refrigerator 540.

The AR platform 520 here may be a wheel-driven go-cart style remotecontrolled car. The AR platform 520 has a camera on its top. However,the camera is not shown because the image seen on the display is createdby the camera mounted on the AR platform 520. A portion of the ARplatform 520 (e.g. the front of the car) is just visible along with thewheels.

The camera of the AR platform 520 here sees a visual landmark 527mounted atop a physical gate that may be placed within the world. Thegate, if used, may be a half circle or ring with distinctive markings,or stickers, or shape; that can be used to operate as a series ofcheckpoints through which the AR platforms may pass through during thecourse of a race or route which may be defined by those gates (e.g.goals to reach three laps, or to traverse through each gate in anparticular order). The visual landmark 527 in this case is a QR code. QRcodes are typically black and white, easily recognizable and readablewith relatively simple computer vision techniques, and may function toact as “gateways” or checkpoints in a racing game or in a traversal gamethroughout a real-world environment. However, other visual landmarks, asdiscussed above, may be used instead of, in addition to, or in place ofthe QR code.

In some cases, visual landmarks will be generated by the objectrecognition software 313 such that no placed visual landmarks arenecessary at all. In those cases, the relative location of, for example,the refrigerator, will be noted as the AR platform traverses the livingroom 570 during a race or traversal. When the AR platform 520 is back tothe place where it began or has reached a location designated based uponobject recognition, the objective may have been reached without the useof any QR code or similar visual landmark.

FIG. 6 is an example display screen of a mobile device 610 showingremote camera augmented reality and race progression tracking. The sameliving room 670 is visible with the same chair 650 and refrigerator 640,but now augmented reality objects are superimposed over some of thephysical objects within the room.

First, the AR platform 620 is visible, but a character 652 and steeringwheel 654 are superimposed over the AR platform 620. This can be done insuch a way that the character appears to be sitting in an augmentedreality seat or cockpit. As the image data and motion data are updated,the character 652 and steering wheel 654 may be animated such that theyoperate in conjunction with the input controls received from the mobiledevice 610 so that they appear to “steer” or otherwise operate the ARplatform 620. The animation may be designed such that the augmentedreality and real AR platform 620 appear to integrate as tightly aspossible while under the control of the mobile device 610.

In addition, the visual landmark 527 is now replaced with an augmentedreality overlay 657 identifying the first checkpoint and providinginformation about an ongoing racing competition. This overlay 657 isonly an example. Overlays may be placed anywhere and may provideinformation or may merely be decorative or aesthetic. The overlay 657can conveniently “hide” the visual landmark 527 of FIG. 5 (shown as a QRcode) from the user and pass along information to the augmented realitydriver. To the extent that visual landmarks are based upon imageprocessing, similar overlays to overlay 657 may be placed in locationsthat make sense for the particular type of augmented reality experience(e.g. flat places or simply floating about marked locations like lapmarkers or checkpoints).

Finally, a leaderboard 658 is displayed as an augmented reality overlaythat may float near the AR platform 620 or that may be fixed in placerelative to the edge of the screen.

FIG. 7 is an overview of a sample race progression tracked using raceprogression tracking. This is intended to be a top-down perspective ofan augmented reality race that may be the race of FIGS. 5 and 6. Here,we see the AR platform 720 in approximately the same position as inFIGS. 5 and 6, with visual landmark 727 in the same location between thechair 750 and refrigerator 740. We see two other visual landmarks 728and 729 which may be used to track progression through an augmentedreality environment through a series of laps or other traversal. Thecamera(s) on the AR platform 720 may transmit image data to a mobiledevice 310 that detects the visual landmarks and identifies a relativelocation of the AR platform 720.

Though the examples of FIGS. 5-7 are provided with respect to a racinggame incorporating augmented reality elements (e.g. the AR racercharacter, steering wheel, lap counters, and checkpoint notifications)other options are envisioned for use with the same overall systems. Forexample, the detected physical characteristics of a location may beconverted into a virtual reality environment with ongoing image datawholly (or only partially) replaced with a computer-generated virtualreality environment. Then, a character may traverse that space whilefollowed by the AR platform 720 acting as a “camera” for the augmentedreality character. This is described more fully below.

Likewise, the race or course could take place in the air using a flyingAR platform 720. Games and experiences other than racing are alsoenvisioned. Augmented reality characters or animals may be generated bya mobile device. The user may use a mobile device to guide the ARplatform 720 through the characters or animals as an augmented reality“tractor” to tend to a farm or following an augmented reality characteras it tends to his or her augmented reality sheepwithin a physical spacein a manner similar to Harvest Moon® type games, but taking place in aplayer's living room or kitchen. The AR platform 720 may, in such cases,be a flying drone or a wheel-based drone that is, itself, augmented witha front-end loader or plow in the associated visual images created bythe mobile device or may follow an augmented reality character orvehicle around within a real, physical space that has been augmented toincorporate animals and plants. An augmented reality pet may live in aspace, closely followed by the AR platform 720 as the pet plays in thereal-world space. A “pen” may be set up using visual landmarks to keepthe pet within a space.

The system may integrate virtual characters or objects and real objectssuch that a platformer type game may take place with an augmentedreality character traversing a physical real-world space while followedby the AR platform 720. In such cases, the physical layout of the spaceshould be detected to enable the platformer style game to take place.The AR platform 720 must be controlled so as to intelligently move aboutthe space and follow the augmented reality character. This example isdiscussed more fully below. These examples and others are envisioned.

The relevant aspect of each of these examples is that an AR platform 720is capable of moving about a space and integrating augmented realityobjects, characters, or elements within that space to enable augmentedreality interactions that do not rely upon a user-worn head mounteddisplay or a user holding an AR viewing screen in their hands whilemoving the physically connected hand-held camera about a space. Instead,the user controls a stand-alone, separate AR platform 720 that movesabout a space and provides the perspective from which image data isreceived, and upon that image data the augmented reality issuperimposed.

FIG. 8 is a flowchart of adjustment of color and lighting for augmentedreality objects to correspond to real world objects. The process beginsat start 805 and ends at end 895, but can take place cyclically orrepeatedly as lighting and other elements change or update while the ARsystem functions.

After the start at 805, the image data is captured at 810. As discussedabove, this is preferably image data captured from the AR platform 320(FIG. 3). The camera(s) 323 of the AR platform 320 may be arranged suchthat generated image data may incorporate at least a portion of one ormore physical objects within the frame of the image data. Those physicalobjects may have known characteristics, such as shapes and colors. Inparticular, an actual pantone color may be known or may be known to bein a particular color range.

As envisioned for at least one of the applications of this technology, aracing game includes a physical AR platform 320 that mounts camera(s)323 so that a portion of the AR platform 320 is visible. This may appearas a “hood” or front of a vehicle, as if it were an actual race car.Alternatively, if the application is a flight simulation, a front-end ofa plane may be physically present on the AR platform 320, but lacking anexpected propeller. The hood of the front of a vehicle may benon-descript, essentially a single color.

The image data received from the AR platform 320 may be analyzed byobject recognition 313 to perform color and lighting detection at 820.The color detection compares the known color or color range expected forthe visible portion of the physical object within the image data. Thelighting detection may indicate that lighting appears to be coming froma particular direction or applying a shadow in a certain direction, orthat there is a reflection in a certain direction and with a certaincolor. As a result of the lighting or other ambient characteristics ofthe environment in which the physical AR platform 320 is operating, thecolor of the physical AR platform 320 may appear differently than ageneric pantone color or color range.

As discussed above, an augmented reality character, or other aspects ofthe AR platform 320 that are not physical may be superimposed over theimage data to increase immersion for users. For example, characters mayreact to control inputs instructing the character to “turn” aremote-controlled AR platform 320. Also, a steering wheel and seat forthe AR platform 320 that are not physically present may be superimposedover the physical AR platform 320 within the image data by the AR/VRsoftware 311 to make it appear as though the AR platform 320 has acharacter sitting on it or within it driving the AR platform 320. Acockpit for a plane or a race car may be superimposed.

Those characters and augmented reality objects that are intended to makeup a portion of the AR platform 320 should match as closely as possibleto the actual color of the physical portion of the AR platform that isvisible on the display 314. Otherwise, the character, cockpit, seat,steering wheel, and the like will appear out of place or disconnectedfrom the AR platform 320 and this will make the experience less-pleasantfor a user or player. For the airplane with no propeller example above,propeller animation may have a particular reflection superimposed overthat propeller (e.g. light from a particular angle) to make it appearcolor and lighting correct.

Therefore, following color and lighting detection at 820, the AR avatar(e.g. a character and any other augmented reality objects) are adjustedat 830 to better match the color and lighting of the physical ARplatform 320 portion visible on the display 314 based upon the imagedata. This may take place in substantially real-time on an ongoing basisas lighting and directionality of the lighting (e.g. shadow casting,reflections, etc.) change as the AR platform 320 moves about a physicalspace.

This may involve creating a virtual shadow, providing a reflection thatmirrors or matches one shown on the front of the AR platform 320 ordarkening, lightening, or providing adjustments to the color warmth ortone of the AR avatar, characters, objects, and the like while the ARplatform 320 is moving about.

The augmented reality overlay is then generated at 840 to correspond tothe detected colors and lighting. The same process may begin again forthe next frame of video or the process may take place only every fewframes to save on processing power. The process is shown ending at 895,but the process may repeat for each frame of video from the image datato ensure that the augmented reality objects, overlays, characters, andthe like correspond to the actual lighting and color tones within thespace the AR platform 320 is operating.

Color correction can be relatively straightforward and may beaccomplished by simply taking a sampling of the average color of thevisible portion of the AR platform 320 and adjusting the color of anyassociated AR overlay such that its average color matches that in theimage data for the AR platform 320. This may be an increase or decreasein one or more of the red, green, blue (RGB) components of a given colorthat may be applied equally to each pixel to reach the appropriate colorrange. In more complex systems, transitions from overlay to physicalobjects (e.g. the borders between physical objects and AR objects) maybe handled on a pixel-by-pixel basis to “smooth” any transitions betweenthe two from a color perspective.

Lighting systems are also known in the space. Directional lighting usingshaders has been known for some time in game engine technology. Morerecently, ray tracing has added the capability for game engine lightingto act in a manner quite similar to actual lighting (e.g. casting lightwhere it would naturally fall based upon a “ray” and casting shadowswhere it would naturally fall based upon another “ray”). Projecting thisbackwards to estimate lighting source locations relative to AR objectsmay be a bit more complex. The best systems rely upon machine learningto determine light source location based upon training from manyhundreds or thousands of images with correct relative light sourcelocations identified. Once sources are identified, even roughly, shadersor ray tracing may be used to project appropriate lighting on augmentedreality objects.

FIG. 9 is a flowchart of augmented reality and virtual reality remotecamera tracking. This process has a start 905 and an end 995, but alsorepeats internally as the process continues. The process may begin eachtime the AR platform's camera(s) are moved or begin operating within anew space. This flowchart describes the way in which the AR platform 320is operated to adjust its associated camera(s) 323 to track an augmentedreality object or character as it moves about in a physical space (e.g.a character moving about as controlled by a user of a mobile device310). This will generally be slightly different than the race carexample above. Here, there may or may not be a visible component of theAR platform 320 within the image data. Instead, the AR platform 320 maymerely act as a real-world tracking camera for an augmented realitycharacter or object.

After the start 905, the attributes of the location may be detected at910. This step is shown as optional, but may be preferable whereavailable. At this step, the AR platform (e.g. AR platform 320 in FIG.3) uses its camera(s) 323 to capture images of the surrounding area.There may be a run-through or a pattern for moving about in a space(e.g. a drone traversing an area) in order to capture sufficient imagesof a location or space to enable detection of the attributes to takeplace. After that, the image data may be passed to the mobile device 310and object recognition 313 may use the captured image data to generate athree-dimensional map of the location. The map may also be based uponinfrared or other depth-related data provided as a part of the imagedata. The map need not be particularly precise, but a generalized layoutmay be helpful as the AR platform 320 moves about so that the ARplatform 320 avoids running into walls and other objects as it moveswithin the location.

The next step is to generate the augmented reality character at 920.Although described as a character, the augmented reality character maybe a car, a tree, a setting, a background, or virtually any other objector set of objects. But, for purposes of this description, it will bepreferable if the augmented reality object is capable of movement, atthe direction of a user of a mobile device 310. As a result, theaugmented realty character will be described as a character for theremainder of the description of this and the following two figures.

The character generated may be, for example, a person who is projectedusing augmented reality into the physical space. The three-dimensionalmap generated at 910 may be helpful in ensuring that the character doesnot appear in the middle of a wall or in the middle of another physicalobject. The AR/VR software 311 may intelligently project the characterinto the location to avoid clipping through actual objects.

The character is generated in images shown on a display, like display314, that are captured by the AR platform 320 as image data andtransmitted to the mobile device 310 for display. The AR/VR software 311may generate the character as an augmented reality object within thephysical space.

Next, user control input may be received at 930. Previously, the usercontrol input adjusted the AR platform 320 itself in the racing example.Here, the user control input is received by the mobile device 310 and isintended to be direction to the augmented reality character itself. Thecontrol input may promptly cause the character to move, though there isno virtual camera to follow this character around in the real world.Instead, the AR platform 320 must keep the augmented reality characterin “frame” of the video.

To aid in this, the AR/VR software 311 may impose restrictions on thespeed of movement or the direction (e.g. only linear movement like aplatformer). In another example, the AR platform 320 may be a flyingdrone that hovers about in the space and can react relatively quickly tochanges in the character's movement. The AR platform 320 may keepsufficient distance and rely upon digital zoom, such that it can trackan augmented reality character as it moves within a physical space. TheAR platform 320's camera(s) 323 may also be capable of panning androtation so as to more easily keep augmented reality characters inframe.

In response to the user control input, the character may be adjusted at940. This adjustment may be merely to cause the character to jump orsquat within the physical world. In another example, this may cause anaugmented reality plane to do a barrel roll or shoot its weapons. Theadjustment may be to move from side to side or to jump onto anotheraugmented reality object or onto a real object in the physical world. Inthe case of the latter, the detected location attributes (e.g.generation of even a crude three-dimensional model of the space) may behelpful to ensure that the augmented reality character can jump on “top”of those objects.

In response to the user control input at 930 and the characteradjustment made at 940, the AR platform is adjusted at 950. As indicatedabove, the AR platform 320 operates to maintain the augmented realitycharacter within the frame of the image data being captured by the ARplatform 320. The AR platform 320 is not directly controlled. As itmoves, it may take into account the attributes of the physical spacedetected at 910. In this way, it can avoid running into walls or otherobjects.

The adjustment at 950 may also take into account any limitations orobjectives imposed by the AR/VR software 311. For example, if theexperience is like a platformer with a goal at one end of the physicalliving room, then the AR platform 320 may anticipate that the characterwill, overall, move toward that goal and not away from it. Therefore,the AR platform 320 may pan to a space behind the character to ensurethat the character remains in frame on its way to the goal.

Though described above as intelligently tracking the augmented realitycharacter, the AR platform 320 may be directly controlled in some cases,though those controls are distinct from those of the augmented realitycharacter. In these cases, separate controls may be used to cause the ARplatform 320 to move relative to the character. For example, themovement requested by a user may not be directly translated, but may allbe completed relative to the augmented reality character so that thecharacter remains in frame and avoids hitting objects or walls in thephysical world.

An individual may wish to have a different view of his or her augmentedreality character and, thus, may instruct the AR platform 320 (and itscamera(s) 323) to move to the left or right or up or down. The ARplatform 320 may respond to those instructions, while rotating thecamera(s) 323 to keep the augmented reality character in frame. In othercases, the camera(s) 323 may be relatively “fixed” so that the user hasno or very limited control relative to his or her character.

Next, additional input may or may not be received at 955. If not (“no”at 955), then the process ends at 995. The process may begin again ifsubsequent movement is detected or if the AR platform 320 or AR/VRsoftware 311 is initialized again.

If additional input is received (“yes” at 955), then the processcontinues. The additional input is preferably input instructing thecharacter to move or adjust, but may be input directly controlling thecamera or may be other input that may require adjustment of thecharacter, the images displayed on the display 314 or the AR platform320.

Next, a determination is made whether there has been (or the requestedinput caused) an AR/VR transition at 965. If not (“AR” at 965), then theprocess continues with adjustment of the character at 940 and adjustmentof the AR platform at 950 as described above.

If an AR/VR transition is made (“VR” at 965), this means that thecontrol input caused the character to transition between an augmentedreality overlay within the real world and a wholly virtual realityenvironment. As discussed above the virtual reality environment is onein which all of its elements are computer generated. This transition maytake place, for example, through an augmented reality “portal” or by adoor-opening transition animation, e.g., a “flash” animation or othersymbolic indication on the display 314 that the augmented realitycharacter being followed by the AR platform 320 in the real world hasrequested transition to or been transported to a virtual reality world.

This virtual world is notable with respect to the present disclosure inthat it cedes responsibility of movement of the AR platform 320 relativeto the character once that transition is made. So, an AR platform 320camera(s) 323 can follow an augmented reality character as describedabove so long as that augmented reality character is operating with thereal, physical world. Once the character transitions to an entirelyvirtual reality, that AR platform 320 need no longer follow thatcharacter at all.

Instead, a virtual camera can move, relative to the character at 970.This virtual camera may take its starting point and position from the ARplatform 320 position relative to the character and other objects. Thevirtual reality may begin, as directed by the AR/VR software 311, from aplace of simply overlaying a complete augmented reality over and aroundthe augmented reality character. Then, when the virtual camera takesover at 970, the AR platform 320 may stop moving, return to a chargingstation (manually or automatically), hover awaiting the augmentedreality character's return from virtual reality, or otherwise becomenon-responsive to user commands.

Unlike the augmented reality character within the physical world, nowthe AR/VR software 311 has to generate the entirety of the world,including the character. So, the world itself must be adjusted at 980 tocorrespond to the movements requested by the user for the character.This may be as simple as updating the world with the distant horizon asa character moves further into the virtual reality environment.Alternatively, it may be rendering portions of the world not visiblepreviously as a user adjusts the virtual reality camera. The characterwithin that world may also be adjusted as directed by the control input.

Next, a determination whether additional input was received at 955 ismade again. If that additional input causes no transition, the processcontinues within VR at 970 and 980 with the camera being virtual.

If an AR/VR transition to AR occurs (“AR” at 965), then the charactermust be adjusted at 940 again within the physical world and the ARplatform 320 must be adjusted at 950. It is preferable if the AR/VRtransitions have some warning so that the AR platform 320 may adjustitself to better match the relative location to the character within thevirtual reality environment before the transition occurs. In that way,the transition can be relatively seamless. Though, other transitions,with perspectives shifting to the real camera of the AR platform 320from the virtual camera of the virtual reality environment to the actualcamera(s) 323 of the AR platform 320 may also take place.

The VR transition may be a relatively seamless way to provide a chargingopportunity for the AR platform 320. The AR platform 320 likely willrely upon battery power and may function only for a limited time withoutcharging again. Transitions to virtual reality can offload all of thecamera tracking to a traditional virtual camera, as is known within gameengines and virtual reality environments. During this time, the ARplatform 320 can return to a base station to charge or replacebatteries. Once the AR platform 320 is ready, it may return and atransition back to an augmented reality combining the real world andaugmented reality characters can begin again.

FIG. 10 is an example of augmented reality remote camera tracking. Here,we see the same living room 1070 we saw previously in FIG. 5. The samechair 1050 and refrigerator 1040 are present.

Here, an AR platform 1020 is a flying drone incorporating at least onecamera. The AR platform 1020 may capture image data (live video) of theliving room 1070 with the AR/VR software 311 (FIG. 3) overlaying anaugmented reality character 1052 a at time t within the scene.

A user may request movement of the augmented reality character 1052 asuch that the user jumps on top of an ottoman 1051 near the chair 1050.The augmented reality character 1052 b (now, at time t+1) is mid-airabout to land on the ottoman 1051. At time t+2, the augmented realitycharacter 1052 c is standing atop the ottoman.

During all of time t, t+1, and t+2; the AR platform 1020 may track theaugmented reality character 1052 a, 1052 b, and 1052 c as though he wereactually in the physical world. The AR platform 1020 may have previouslyscanned the living room, as discussed above, to have a general layout ofthe three-dimensional environment, and may pass that information alongto the AR/VR software 311 so that the character 1052 c is able to standatop the ottoman 1051 because the AR/VR software 311 is aware that it isa three-dimensional object that sits a foot or two above the floor. TheAR platform 1020 may track the augmented reality character 1052 a, 1052b, and 1052 c as it moves about in the physical space, intelligently ormanually adjusting to the location of that character 1052 a, 1052 b, and1052 c as movements are made.

FIG. 11 is an example of augmented reality into virtual reality remotecamera tracking. The same living room 1170, with the same chair 1150 andottoman 1151 is shown. The refrigerator (1040 in FIG. 10) is missingbecause that area is being generated in virtual reality 1180.

The augmented reality character 1152 a is shown in front of a portal1153. As discussed above, various transitions are possible, the portal1153 is only one example. Transitions may be fade-to-black followed byfade-up in a virtual reality environment, or may be seamless such asportals, doors, windows, or slow transitions from reality to virtualreality.

At time t, the augmented reality character 1152 a is within the livingroom 1170. The AR platform 1120 is tracking the movement of theaugmented reality character 1152 a as it moves within the living room1170.

At time t+1, the augmented reality character 1152 b has moved intovirtual reality 1180 through the portal 1153 as directed by a user of amobile device 310. The virtual reality may have other characters, likefictional character 1155, present and may be entirely rendered by theAR/VR software 311. The user may elect to instruct the augmented realitycharacter 1152 b to transition back to reality.

During the time within the virtual reality 1180, the AR platform 1120may merely hover awaiting further augmented reality character 1152 amovement within the physical living room 1170, sit still, or return to acharging dock or land to conserve power.

Closing Comments

Throughout this description, the embodiments and examples shown shouldbe considered as exemplars, rather than limitations on the apparatus andprocedures disclosed or claimed. Although many of the examples presentedherein involve specific combinations of method acts or system elements,it should be understood that those acts and those elements may becombined in other ways to accomplish the same objectives. With regard toflowcharts, additional and fewer steps may be taken, and the steps asshown may be combined or further refined to achieve the methodsdescribed herein. Acts, elements and features discussed only inconnection with one embodiment are not intended to be excluded from asimilar role in other embodiments.

As used herein, “plurality” means two or more. As used herein, a “set”of items may include one or more of such items. As used herein, whetherin the written description or the claims, the terms “comprising”,“including”, “carrying”, “having”, “containing”, “involving”, and thelike are to be understood to be open-ended, i.e., to mean including butnot limited to. Only the transitional phrases “consisting of” and“consisting essentially of”, respectively, are closed or semi-closedtransitional phrases with respect to claims. Use of ordinal terms suchas “first”, “second”, “third”, etc., in the claims to modify a claimelement does not by itself connote any priority, precedence, or order ofone claim element over another or the temporal order in which acts of amethod are performed, but are used merely as labels to distinguish oneclaim element having a certain name from another element having a samename (but for use of the ordinal term) to distinguish the claimelements. As used herein, “and/or” means that the listed items arealternatives, but the alternatives also include any combination of thelisted items.

It is claimed:
 1. A system for remote camera augmented realitycomprising: a remote computing device, including a processor andnon-volatile memory, separate from a movable augmented reality platform,comprising: the processor which is programmed to: perform imageprocessing on image data to identify visual landmarks within a physicalspace, the image data captured while a movable augmented realityplatform moves through the physical space; define a course through thephysical space, the course defined by a path within the physical spacein relation to the visual landmarks and generated by movement of themovable augmented reality platform within the physical space; and adisplay for displaying the image data with augmented reality mediaoverlaid while the movable augmented reality platform moves along thecourse as controlled by instructions from the remote computing device.2. The system of claim 1, further comprising: the movable augmentedreality platform including: a camera for capturing the image data of anarea in front of the camera on the movable augmented reality platform;and a locomotive system for moving the movable augmented realityplatform within the physical space in response to instructions receivedfrom the remote computing device.
 3. The system of claim 1 wherein thevisual landmarks comprise high-contrast machine-readable images, visiblewithin the image data as the movable augmented reality platform moveswithin the physical space.
 4. The system of claim 1 wherein the visuallandmarks comprise a three-dimensional model of the area surrounding thephysical space generated by the remote computing device as a result ofthe image processing.
 5. The system of claim 1 wherein the imageprocessing is reliant upon the image data from the movable augmentedreality platform and from at least one other movable augmented realityplatform operating within the physical space.
 6. The system of claim 1wherein the movable augmented reality platform communicates with atleast one other movable augmented reality platform operating within thephysical space to transmit the image data and to receive image data fromthe other movable augmented reality platform.
 7. The system of claim 1wherein the movable augmented reality platform communicates with atleast one other movable augmented reality platform to provide motiondata captured by the moveable augmented reality platform that is used inconjunction with the image data to define the course.
 8. The system ofclaim 1 wherein the movable augmented reality platform performs imageprocessing of the image data to generate preprocessed spatial data andprovides the preprocessed spatial data to at least one other augmentedreality platform operating within the physical space.
 9. A method ofremote camera tracking comprising: performing image processing on imagedata to identify visual landmarks within a physical space, the imagedata captured while a movable augmented reality platform moves throughthe physical space; defining a course through the physical space, thecourse defined by a path within the physical space in relation to thevisual landmarks and generated by movement of the movable augmentedreality platform within the physical space; and displaying the imagedata with augmented reality media overlaid as the movable augmentedreality platform moves along the course while controlled by instructionsfrom the remote computing device.
 10. The method of claim 9 furthercomprising: capturing the image data of an area in front of a camera onthe movable augmented reality platform; and moving the movable augmentedreality platform within the physical space using a locomotive system inresponse to instructions received from the remote computing device. 11.The method of claim 9 wherein the visual landmarks comprisehigh-contrast machine-readable images, visible within the image data asthe movable augmented reality platform moves within the physical space.12. The method of claim 9 wherein the visual landmarks comprise athree-dimensional model of the area surrounding the physical spacegenerated by the remote computing device as a result of the imageprocessing.
 13. The method of claim 9 wherein the image processing isreliant upon the image data from the movable augmented reality platformand from at least one other movable augmented reality platform operatingwithin the physical space.
 14. The method of claim 9 wherein the movableaugmented reality platform communicates with at least one other movableaugmented reality platform operating within the physical space totransmit the image data and to receive image data from the other movableaugmented reality platform.
 15. The method of claim 9 wherein themovable augmented reality platform communicates with at least one othermovable augmented reality platform to provide motion data captured bythe moveable augmented reality platform that is used in conjunction withthe image data to define the course.
 16. The method of claim 9 whereinthe movable augmented reality platform performs image processing of theimage data to generate preprocessed spatial data and provides thepreprocessed spatial data to at least one other augmented realityplatform operating within the physical space.
 17. A system for remotecamera augmented reality comprising: a remote computing device,including a processor and non-volatile memory, separate from a movableaugmented reality platform, comprising: the processor which isprogrammed to: perform image processing on image data to identify visuallandmarks within a physical space, the image data captured while amovable augmented reality platform moves through the physical space;define a course through the physical space, the course defined by a pathwithin the physical space in relation to the visual landmarks andgenerated by movement of the movable augmented reality platform withinthe physical space; and a display for displaying the image data withaugmented reality media overlaid while the movable augmented realityplatform moves along the course as controlled by instructions from theremote computing device; and the movable augmented reality platformcomprising: a camera for capturing the image data of an area in front ofthe camera on the movable augmented reality platform; and a locomotivesystem for moving the movable augmented reality platform within thephysical space in response to instructions received from the remotecomputing device.
 18. The system of claim 17, wherein the remotecomputing device further comprises a user interface for generating theinstructions for control of movement of the movable augmented realityplatform through the physical space.
 19. The system of claim 17, whereinthe movable augmented reality platform communicates with at least oneother movable augmented reality platform operating within the physicalspace to transmit the image data and to receive image data from theother movable augmented reality platform.
 20. The system of claim 1wherein the movable augmented reality platform performs image processingof the image data to generate preprocessed spatial data and provides thepreprocessed spatial data to at least one other augmented realityplatform operating within the physical space.